1. Field of the Invention
The present invention generally relates to post-processing of block-based coded video, and more particularly to a filtering method and apparatus for removing blocking artifacts from MPEG-coded video.
2. Description of the Prior Art
A wide range of new applications in visual communications have been made possible due to the rapidly evolving telecommunication and computer industries. Almost all the related applications, including video conferencing, mobile or PSTN video telephony, video over the Internet, and digital TV broadcasting, require very efficient data compression methods to fit a large amount of visual information into the narrow bandwidth of communication channels while preserving acceptable quality of the reconstructed data.
To efficiently compress a time variable video sequence, redundancy in the temporal domain as well as in the two dimensional spatial domain must be reduced. The Moving Picture Experts Group (MPEG) standards use a discrete cosine transform (DCT) to reduce the redundancy in the two dimensional spatial domain and a motion compensation method to reduce the redundancy in the temporal domain.
The DCT is a method of reducing the correlativity between data through a two dimensional spatial transformation. Each block in a picture is spatially transformed using the DCT after the picture is divided into blocks. Data that has been spatially transformed tends to be driven to a certain direction. Only a group of the data driven in the certain direction is quantized and transmitted.
Pictures, which are consecutive in the temporal domain, form motions of a human being or an object at the center of the frame. This property is used to reduce the redundancy of the temporal domain in the motion compensation method. A volume of data to be transmitted can be minimized by taking out a similar region from the preceding picture to fill a corresponding region, which has not been changed (or has very little change), in the present picture. The operation of finding the most similar blocks between pictures is called a motion estimation. The displacement representing a degree of motion is called a motion vector. MPEG uses a motion compensation-DCT method so that the two methods combine.
When a compression technique is combined with a DCT algorithm, the DCT transform is usually performed after input data is sampled in a unit size of 8×8, and the transform coefficients are quantized with respect to a visual property using quantization values from a quantization table. Then, the data is compressed through a run length coding (RLC). The data processed with the DCT is converted from a spatial domain to a frequency domain and compressed through the quantization with respect to the visual property of human beings, not to be visually recognized. For example, since the human eye is insensitive to high frequencies, a high frequency coefficient is quantized using a large step size. Thus, a quantization table is made according to external parameters, such as a display characteristic, watching distance, and noise, to perform an appropriate quantization.
For the quantized data, the data having a relatively high frequency is coded with a short code word. The quantized data having a low frequency is coded with a long code word. Thus, the data is finally compressed.
FIG. 1 shows a plurality of adjacent 8×8 pixel blocks 100, 102, 104 used according to the above described MPEG compression. In processing a moving picture as discussed above, these blocks 100, 102, 104 are individually processed to maximize the compression ratio and coding efficiency. However, this individual processing causes blocking artifacts that are noticeable at boundaries between blocks. Both horizontal block boundaries 106 and vertical block boundaries 108 are present and tend to cause square patterns (blocking artifacts) that are quite apparent to the human eye.
At low coding bit rates, blocking artifacts are a serious problem for moving picture compression. Since a realtime operation is necessary in coding and decoding a moving picture, it is difficult to reduce the blocking artifact with a small operation capacity. As such, existing video compression algorithms and standards continue to have unwanted visual artifacts appear when the original information is compressed with loss, and these unwanted visual artifacts become more visible when the coding bit rates become lower.
In the traditional block-based video compression standards such as MPEG-1 and MPEG-2, the blocking artifact is the most noticeable artifact. In actuality, there are two main sources of blocking artifacts. As mentioned previously, the major source comes from individually encoding each 8×8 DCT block without considering the correlation between adjacent blocks. In this case, coarse quantization of the transform coefficients causes discontinuities at the block boundaries. The second source of blocking artifacts is from the motion compensated prediction. Motion compensation involves coding pixel data from a reference frame and results in a discontinuity at the edge of the copied block. Additionally, any blocking artifacts existing in the reference frame may also be propagated to a current frame due to copying process. As such, the blocking artifacts resulting from motion compensated prediction can occur at any position inside an 8×8 predictive block.
Accordingly, various methods for reducing blocking artifacts in coding systems that individually processes blocks have been developed. For example, de-blocking filters can be integrated into a video codec as either a loop filter or a post filter. Loop filters operate inside the motion compensation loop so that the filtered frames are used as reference frames of subsequent coded frames. However, this method of changing the processes of coding and decoding increases the amount of bits to be transmitted. Additionally, such an approach is unacceptable in many situations because the resulting data stream no longer complies with the applicable standards. Unlike loop filters, post filters operate on the decoded bit stream outside the coding loop. Thus, post filters can operate in conjunction with the original video compression standards. In other words, post filters can be easily integrated into any standard decoder using various implementation methods designed independent of the standard. A great amount of work has been done in the field of de-blocking filters, and the following papers are incorporated herein by reference:
[1] R. Rosenholtz and A. Zakhor, “Iterative procedures for reduction of blocking effects in transform image coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 2, pp. 91-95, March 1992.
[2] Y. Yang, N. P. Galatsanos, and A. K. Katsaggelos, “Regularized reconstruction to reduce blocking artifacts of block discrete cosine transform compressed image,” IEEE Trans. Circuits Syst. Video Technol., vol. 3, pp. 421-432, December 1993.
[3] Y. Yang, N. P. Galatsanos, and A. K. Katasaggelos, “Projection-based spatially adaptive reconstruction of block-transform compressed images,”IEEE Trans. Image Processing, vol. 4, pp. 896-908, July 1995.
[4] T. Meier, K. N. Ngan, and G. Grebbin, “Reduction of blocking artifacts in image and video coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 9, pp. 490-500, April 1999.
[5] T. P. O“Rourke and R. L. Stevenson, “Improved image decompression for reduced transform coding artifacts,”IEEE Trans. Circuits Syst. Video Technol., vol. 5, pp. 490-499, December 1995.
[6] H. W. Park and Y. L. Lee, “A postprocessing method for reducing quantization effect in low bit-rate moving picture coding,”IEEE Trans. Circuits Syst. Video Technol., vol. 9, pp. 161-171, February 1999
[7] S. D. Kim, J. Yi, H. M. Kim, and J. B. Ra, “A deblocking filter with two separate modes in block-based video coding”, IEEE Trans. Circuits Syst. Video Technol., vol. 9, pp. 156-160, February 1999
[8] P. List, A. Joch, J. Lainema, G. Bjontegaard, and M. Karczewicz, “Adaptive deblocking filter,” IEEE Trans. Circuits Syst. Video Technol., vol. 13, pp. 614-619, July 2003.
There are two approaches commonly adopted to reduce blocking artifacts: an iterative approach (references [1-5]) and a non-iterative approach (references [6-8]). Several iterative de-blocking methods have been used to alleviate blocking artifacts, including projection onto convex sets (POCS) (references [1-3]), Markov random field (MRF) (reference [4]), and maximum a posteriori (MAP) restoration (reference [5]). The main drawback of these iterative de-blocking methods is their high computational complexity, which prevents them from being used in a realtime video decoder.
Conversely, especially for real time applications, noniterative de-blocking methods mostly using adaptive filters have also been adopted to reduce blocking artifacts. Park et al. (reference [6]) define a blocking semaphore to represent the strength of the blocking artifacts in the MPEG-4 decoder. The blocking semaphore is extracted from the DCT domain of each 8×8 block in the intra video object plane (VOP). The semaphores of the inter-VOP are calculated from both the residual signal and the semaphores of the reference VOP. Then, a one-dimensional strong low-pass filter and a special type of weak filter are adaptively applied on the horizontal and vertical block boundary to the blocking semaphores.
FIG. 2 shows a block diagram showing the mode decision procedure for a de-blocking filter having two separate filtering modes as proposed by Kim et al. (reference [7]). This de-blocking filter has also been included as an information section in the MPEG-4 video standard. The proposed filter performs one-dimensional filtering along the horizontal (and vertical) block boundaries 200 between 8×8 pixel blocks. In step 202, local image characteristics around the block boundary 200 are examined to determine if the block boundary 202 is a smooth region. In steps 204 and 206, an assessment is performed to determine if the block boundary represents a real edge that should be present in the image. If the block boundary is not a real edge, for the smooth region mode 212 to which flat regions belong, a strong nine-tap smoothing filter is applied to pixels inside the blocks as well as to the pixels at the block boundary. More specifically, the strong nine-tap filter is applied to pixels v1 to v8. For the default region mode 208, to which complex regions belong, a frequency analysis technique using 4-pt DCT kernel is used to filter pixels v4 and v5 at the block boundary only. If the block boundary 200 represents a real edge (steps 210 and 214), no filtering is applied. The de-blocking method proposed by Kim et al. results in high subjective quality of decoded images at low bit rates; however, it smoothes real edges and severely degrades the objective quality at higher bit rates due to the strong low-pass filter used in the smooth region mode. Moreover, this method does not consider block artifacts in inter-coded MBs that are caused by the motion compensated prediction.
List et al. (reference [8]) propose a sophisticated in-loop de-blocking filter for the H.264/MPEG-4 AVC video coding standard. The proposed in-loop de-blocking filter performs one-dimensional filtering along the block boundaries of a 4×4 block, which is the basic coding unit of the H.264 video codec. The 4×4 block boundaries are assigned boundary-strength (Bs) parameters from 0 to 4. Block boundaries having a B equal to 0 are not filtered, and a special type of weak filter is applied to block boundaries having Bs from 1 to 3. Finally, a stronger low-pass filter is applied to block boundaries having a B equal to 4. The in-loop de-blocking method proposed by List et al. improves both the subjective and objective quality, especially at low bit rates. However, due to its highly adaptive nature, this method also has a very high computational complexity. The filter can account for one-third of the computational complexity of an H.264/MPEG-4 AVC decoder even after a tremendous effort has been made in speed optimization of the filtering algorithms.
Accordingly, due to the above limitations, most of the prior art is only suitable for de-blocking filtering at low bit rates and is unable to provide satisfactory results at higher bit rates. However, the demand for de-blocking filtering of MPEG-encoded video at medium to high bit rates is increasing due to the wide acceptance of MPEG applications, such as digital versatile discs (DVDs) and digital TV.